Vector Search Techniques

Additional Database Vector Querying Details

The Database Guides section to the left has specific details discussing vector search and querying techniques for each supported database. This document covers the general use of the embeddings() macro function. Its use can be applied to the various Qarbine supported databases.

Useful Macro Functions

There are a few macro functions related to AI Assistant usage. The one most relevant to queries is described below.

vector = embeddings(string, aiAssistantAlias)

Obtain a vector list for the given string. The alias specifies which Qarbine configured AI service to use. A null means to use the default.

Using Embeddings for Searches

Overview

An embedding is a way of representing data as points in n-dimensional space so that similar data points cluster together. The typical representation is a list of floating point numbers. This is sometimes referred to as a “vector”. Thus, the phrase “vector search”. Database developers are responsible for creating vector indices on identified appropriate fields. These indices can then be used to retrieve data which is similar (or ‘close’) to a given vector.

The closeness can be determined in several manners such as Euclidean distance, cosine distance, and approximate nearest neighbor (ANN). Euclidean distance measures the straight-line distance between two points in a multidimensional space, cosine distance calculates the angle between two vectors, ignoring their magnitudes, and Approximate Nearest Neighbor is a technique used to efficiently find the "closest" data points to a query point in a large dataset, often utilizing distance metrics like Euclidean or cosine distance, but with a slight trade-off in accuracy for speed.

The Qarbine administrator must define the AI Assistant settings in order to dynamically obtain embeddings. That task is documented separately in an Administration document.

Sample Use of Macro Functions

The macro function to obtain an embedding for a given string is again

vector = embeddings(string, aiAssistantAlias)

This function makes sense to use as part of a query rather than for presenting the possibly large list of floating point numbers in a report via a report template. For ease of reviewing, shown below is using this function in the Data Source Designer tool.

  

Depending on the configuration set by the Qarbine Administrator a possibly large list of numbers is returned.

Note that the default maximum rows for testing in the Data Source Designer is 25. Set that value to zero to indicate to return all of the values. You can use this simple query to obtain embeddings for arbitrary strings that can then be copied and pasted for use in other areas. The array of values can be placed into the clipboard using the options shown below.

  

Sample Query Usage

One use case is to prompt the user for a string to then locate objects in the database which have similar values. The input string is passed to embeddings() and the list result is used in a search query,

A discussion of a CosmosDB’s vector search example can be found at
https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb/vcore/vector-search
That example only uses 3 dimensions for its vector. Many embeddings contain many more than 3 elements. The macro function sampleEmbeddings() returns the list referenced in that article,

vectorContent: [ 0.51, 0.12, 0.23 ]

A sample Data Source query definition when using the CosmosDB implementation of vector search is shown below.

var queryVector = [! embeddings(@userInput) !];
db.exampleCollection.aggregate([
  {
    $search: {
      "cosmosSearch": {
        "vector": queryVector ,
        "path": "vectorContent",
        "k": 4
      },
    "returnStoredSource": true }},
  {
    "$project": { "similarityScore": {
           "$meta": "searchScore" },
            "document" : "$$ROOT"
        }
  }
]);

The “[! expression !]” syntax is used in this example. The “@variableName” approach can also be used for simple query variable substitution. The same technique can be used for other database vector search implementations.

The userInput value can be passed in or obtained from a Qarbine prompt. We then obtain the embeddings for that string and use the result to build the final query. In this case the line

var queryVector = [! embeddings(@userInput) !];

is dynamically adjusted prior to execution to look something like

var queryVector = [0.453, 0.9123, 0.6123, 0.51, … ];

Note the trailing semicolon which is required for multi-line queries.

It is important to remember that the vectors emitted from the embeddings() macro function must use the same model as used to set the target database objects’ fields. If they do not then an error message such as “expected 3 dimensions, not 1536” will be encountered.

A sample answer set for the query above is shown below.

  

Notice the document field contains the primary values of interest. If we insert the first line of

#pragma pullFieldsUp document

The answer set is now

  

For template formulas the removal of the added “document” field layer can simplify things. Rather than #document.bio you can just use #bio.

MongoDB Sample Data

MongoDB Atlas provides a convenient utility to load several sample data sets. An overview can be found at https://www.mongodb.com/docs/atlas/sample-data. One of the data sets is the original MongoDB mflix data set enhanced to have embeddings. For details see
https://www.mongodb.com/docs/atlas/sample-data/sample-mflix/

The embedded_movies collection contains information about movies in the Western, Action, and Fantasy genres from sample_mflix.movies collection with an added plot_embedding field. The embeddings were created using OpenAI's text-embedding-ada-002 embedding model that you can use to query with Atlas Search vector search.

Support for the '$vectorSearch' aggregation pipeline stage is available with MongoDB Atlas 6.0.11, 7.0.2, and above.

A tutorial discussion on this topic can be found at
https://www.mongodb.com/developer/products/atlas/semantic-search-mongodb-atlas-vector-search/

A sample query is shown below.

var queryVector = [! embeddings(@userInput) !];
db.embedded_movies.aggregate([
  {"$vectorSearch": {
    "queryVector": queryVector,
    "path": "plot_embedding",
    "numCandidates": 100,
    "limit": 5,
    "index": "moviesPlotIndex",
      }}

Running this in the Data Source Designer results first in a prompt.

  

Enter some words and click OK. The query is run and the results shown.
To run again press down on CONTROL and click the run button   .

The embedding for the words is retrieved and then used in the query.

Notes on Other Databases

Generally both the “[! … !]” and “@variable” techniques can be used as part of the querying specification for any supported Qarbine database endpoint. For example, here is a Euclidean distance search in Postgres.

SELECT * FROM items 
ORDER BY embedding <-> [! embeddings(@userInput) !] 
LIMIT 5

Here is a Cosine similarity search.

SELECT * FROM items 
ORDER BY embedding <=> [! embeddings(@userInput) !]
LIMIT 5

Notice the “<->” and “<=>” operator differences in the Postgres syntax.